Papers with visual description
ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)
Copied to clipboard
Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell
| Challenge: | Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms. |
| Approach: | They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations. |
| Outcome: | The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps. |
ReLoop: “Seeing Twice and Thinking Backwards” via Closed-loop Training to Mitigate Hallucinations in Multimodal understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination mitigation rely on external verification or post-hoc correction, lacking internal mechanism to validate outputs directly during training. |
| Approach: | They propose a unified closed-loop training framework that encourages multimodal consistency for cross-modal understanding in MLLMs. |
| Outcome: | The proposed framework encourages multimodal consistency for cross-modal understanding in MLLMs. |
Distribution Aware Metrics for Conditional Natural Language Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for conditional natural language generation rely on pairwise comparisons between a single generated text and the best-matching reference. |
| Approach: | They propose a family of meta-metrics that build on existing pairwise distance functions to evaluate conditional natural language generation models. |
| Outcome: | The proposed method evaluates the ability of a model to generate text matching diversity in references in visual description and summarization. |
IC3: Image Captioning by Committee Consensus (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, image captioning models are trained to generate a single “best’ (most like a reference) image caption. |
| Approach: | They propose a method to generate a single caption that captures high-level details from several annotator viewpoints. |
| Outcome: | The proposed method outperforms baseline SOTA models and improves the performance of automated recall systems by up to 84%. |